1. Field of the Invention
The invention relates generally to the field of graphics. More specifically, the invention relates to the field of digital imaging systems and software.
2. Description of the Related Art
In some digital cameras and imaging devices, the color information of a scene is captured by sub-sampling an image typically into three (Red, Green and Blue) color planes. In such cases, the original xe2x80x9craw imagexe2x80x9d is captured with each xe2x80x9craw image pixelxe2x80x9d composed of only one of three vital color/intensity components: either R, G, or B (Red, Green, or Blue). But for certain image rendering devices (such as a monitor or printer), full color resolution, wherein each pixel is composed of a combination of R, G and B components, is desirable. Recovery of full color information from a sub-sampled image of single-color component raw image pixels is popularly known as color interpolation.
In digital still and video cameras and certain other imaging devices, raw images are first represented as rectangular row and column of pixels with each pixel having the intensity value of a particular color plane only. In the case of RGB (Red, Green and Blue) sub-sampling imaging devices, images are obtained and stored in a particular pattern. FIG. 1(a) shows one such pattern, which is considered in describing the invention, known as the xe2x80x9cBayerxe2x80x9d pattern. The Bayer pattern (see FIG. 1(a)), when three color planes are sub-sampled, is characterized by including on a first row of an image, alternating pixels of Red and Green, and then on a next row alternating pixels of Blue and Green. Thus, the Bayer pattern for the first four rows of pixels (with each pixel typically an 8-bit value) is as follows (with the rows thereafter repeating the same pattern):
G R G R G R . . .
B G B G B G . . .
G R G R G R . . .
B G B G B G . . .
xe2x80xa2
xe2x80xa2
xe2x80xa2
As a result, each pixel location contains a value for a single color only. Assuming, as is the case in some imaging devices, that each pixel of the Bayer pattern has 8 bits of resolution (i.e., the pixel is a value representing at a particular location in the captured image the intensity of the color ranging from 0-255), then a xe2x80x9cfull colorsxe2x80x9d pixel, one having all three R, G and B components, would be a 24-bit value. To recover the two missing color components for each pixel, various methods of color interpolation or color recovery have been employed.
In color interpolation, an attempt is made to combine in some manner neighboring pixel(s) to reconstruct, recover, or predict missing color component information for a given raw image pixel. Several prior art interpolation methods are described in xe2x80x9cInteractions Between Color Plane Interpolation and Other Image Processing Functions in Electronic Photographyxe2x80x9d, James E. Adams, Jr., SPIE, (The International Society for Optical Engineering), Vol. 2416 (Electronic Imaging ""95), pages 144-155 (July 1995), with most of these employing pixel averaging techniques to recover missing color components. One notable exception to averaging-type technique are techniques based upon the median computation. The median computation can be described as follows. Consider an ordered set of values {X1, X2, X3 . . . Xn} such that X1xe2x89xa6X2xe2x89xa6. . . Xn. The function median {X1, X2, X3 . . . Xn} would return a value M such that one-half of the values in the set are ranked below that value and the other half are ranked above. Thus, if N is odd, then median {X1, X2, X3 . . . Xn} is   x            N      +      1        2  
and if N is even, then median {X1, X2, X3 . . . Xn} is the value             (                        x                      N            2                          +                  x                                    N              2                        +            1                              )        /    2    ,
or the average of the two middle values in the rank ordered set. By definition, the median computation involves sorting. An unordered set of values will hereinafter be designated by xe2x80x9c()xe2x80x9d whereas an ordered set will be designated as xe2x80x9c{}xe2x80x9d.
When used in color recovery/interpolation, the median computation is used to select a value that more or less is actually present in the neighborhood of the pixel under consideration. By contrast, averaging can have the effect of introduction a value of a color component that is not present. The median computation is one way therefore to reduce noise introduced through color interpolation.
A median technique would interpolate the Red component of the captured Green pixel at row 2, column 2, (referring to the 8-bit Bayer pattern above) as the median of the two nearest red pixels, i.e., Red pixel at row 1, column 2 and the Red pixel at row 3, column 2. In this case, since only two pixels are involved, the median computation reduces to a simple average without need for sorting. Likewise, the Blue component of the same captured Green at row 2, column 2 would be interpolated under an median technique as the median of the two nearest Blue pixels, i.e., the pixel at row 2, column 1 and the Blue pixel at row 2, column 3. The Blue and Red interpolated components are then combined with the original captured Green value of the pixel, row 2, column 2 to form a new pixel which is a composite xe2x80x9cfullxe2x80x9d color pixel. The resultant xe2x80x9cRGBxe2x80x9d full color pixel is an approximation or prediction of the original object/one""s true color at that location in the captured image. FIG. 1(a) illustrates that to store the color interpolated image, three times the memory size of the original Bayer pattern image, would be required, assuming that each color plane component, R, G and B retains the same number of bits after interpolation.
Once each pixel of the image is at a full RGB color resolution (either through color interpolation or by virtue of having been captured at full color resolution), another process known as color space conversion may be performed. Depending on the rendering device or particular application for which the full color image is destined, the RGB xe2x80x9ccolor spacexe2x80x9d (of an interpolated image, see FIG. 1(a)) may need to be converted to a different color space. Certain imaging applications such as image editors or rendering devices such as printers may not use RGB color space in manipulating or rendering images. Instead they may use different color spaces which may have the advantage of being more standardized or better representative of the application/device nature and characteristics. The RGB color space primarily indicates the levels of R, G and B in a pixel location and thus is a measure of position in the color spectrum. An RGB color space image such as one resulting from color interpolating a Bayer pattern image also has both xe2x80x9cchrominancexe2x80x9d and xe2x80x9cluminancexe2x80x9d information although this information is not easily decodable. Luminance refers generally to the level of brightness in an image or pixel, whereas chrominance refers to the hue or placement of color in the color spectrum and saturation or intensity of that hue. In an RGB image, the Green color plane associated pixels contain most of the luminance information while Blue and Red associated pixels contain most of the chrominance information. This is due in part to the response of the human eye to recognize hue variations in the color Green over the colors Red and Blue, which are more associated in the human vision system with brightness or luminance levels. While a fully color interpolated RGB color space is excellent at representing the colors in an image, it is also dependent on the perception of what constitutes Red, Green, and Blue. Based on the responses of the human visual system to Red, Green, and Blue, a standardized color space has been defined known as CIE (Commission Internationale de l""Eclairage) XYZ.
CIE XYZ is a color system standardized by the CCIR (International Consultative Committee on Broadcasting) and is also referred to as the CCIR 601-1 color space. CIE XYZ defines color image information in terms of a first primary (Y component) and two other primary components X and Z. Any color can be represented by a combination of X, Y, and Z values. Roughly speaking, each component X, Y, and Z is a linear combination of the R, G, B primaries. In color space conversion from 24-bit RGB to 24 CIE XYZ, each pixel location has all three components X, Y, and Z. If each component is 8 bits in resolution, then each pixel has a total of 24 bits of information. FIG. 1(b) illustrates the processing step where the 24-bit RGB space is converted into a 24-bit CIE XYZ space. The CIE XYZ was developed in an attempt to eliminate dependence on how Red, Green and Blue are represented in one device as opposed to another. CIE XYZ removes, in part, dependence on device characteristics in representing color, and thus serves to standardizing the content of an image from one device to another.
Traditionally, the process of color interpolation and color space conversion have been independently performed. First, a raw image is color interpolated. When it is stored or transferred it may also be first compressed or encoded. The process of color space conversion ordinarily occurs in software or in an application separate from the imaging system performing the image capture. Often, even color interpolation is performed in software.
Color interpolation and color space conversion also both involve a set of computations. In the case of color interpolation, these computations may constitute equal-weight averaging, according to one traditional color interpolation technique, of neighboring raw image pixels. In the case of color space conversion a transformation matrix is utilized to convert each pixel from an RGB color space value to an XYZ value. The transformation matrix is multiplied by the R,G and B values to generate X, Y and Z converted values.
It would be desirable to design and provide a technique that integrates the operation of color interpolation and color space conversion into a single operation. Such a technique would be advantageous in a digital camera where color interpolation is performed on-camera, such that color space conversion may also be performed in an integrated manner to reduce the overall computational complexity and time.
What is disclosed is a method including the steps of (1) providing an integrated median computation based color space conversion and color interpolation technique, and (2) applying that technique to pixels of a raw image, the raw image pixels without full color resolution, the technique generating therefrom a color interpolated image in a XYZ color space.